Antiferromagnetic topological insulator state in the correlated 
Bernevig-Hughes-Zhang model 
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We study the eflects of electron correlations on the topological phase transition in the Bernevig- 
Hughes-Zhang model using the variational cluster approach where the short-range spatial corre- 
lations are taken into account exactly. We calculate the spin Chern number and local magnetic 
moment to show that the topologically nontrivial antiferromagnetic order exists and that the mag- 
netic transition is of the second order. We furthermore demonstrate that under the spin-quantized 
condition the topological phase transition is caused by the closing of the bulk band gap. 
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I. INTRODUCTION 

Presence of the topologically ordered phases^^ has re- 
ceived much attention in recent years from both theoreti- 
cal and experimental sides. The existence of gapless edge 
states and their characteristic electromagnetic responses 
detected in, e.g., the quantum Hall systems'^''* have been 
understood in terms of the topological orders and their 
unconventional insulator states distinguished from the 
simple insulator states by their nontrivial topological 
structure have been broadly referred to as the topologi- 
cal insulators (TIs)j^— Experimentally, the Z2 topologi- 
cal band insulators (TBIs),^"^^ which are caused by the 
spin-orbit coupling without external magnetic field, have 
been realized in two- and three-dimensional systems such 
as HgTe/CdTe quantum well a-'^^d^ and bismuth based 
compounds.—"— Their gapless edge states are protected 
by the time-reversal symmetry and their topology is re- 
flected in the parity of the number of the edge statesi^^riS 
The relation between the crystallographic symmetry of 
the system and the presence of topologically ordered 
states in such systems has thus been studied intensively 
in recent years 1^— 

The studies of the TBIs have been made within the 
noninteracting band theory on one hand. However, the 
introduction of electron correlations to the TBIs has on 
the other hand attracted great attention from the view- 
point of the possible realization of exotic topological 
phasesi^"— In recent studies, a number of candidates 
for the exotic topologically ordered phases, where both 
the spin-orbit interaction and electron correlations play 
essential roles, have been proposed in transition-metal 
oxides, such as iridium oxides 1^1^ For such systems, the 
properties of the TBIs, which reflect the effects of elec- 
tron correlations and competition between the antiferro- 
magnetic (AF) and topological orders, have been clarified 
in a wide variety of numerical methods, such as varia- 
tional Monte Carlo (VMC) methodrS, quantum Monte 
Carlo (QMC) method)^ dynamical mean-field theory 
(DMFT) jSi,-Ji variational cluster approach (VCA),^ and 
cluster dynamical mean-field theory (CDMFT).'^^ In par- 
ticular, in the DMFT study^^ of the Bernevig-Hughes- 



Zhang (BHZ) model^^ reproducing the TBI state, the co- 
existence of the TBI and axial antiferromagnetic Mott in- 
sulator (AFMI) state has been confirmed by including the 
on-site Hubbard interaction, the state of which is called 
the antiferromagnetic topological insulator (AFTI). In 
the QMC study21 of the Kane-Mele model^i^ another 
model reproducing the TBI state, it was found that the 
topological phase transition between the TBI and in- 
plane AFMI also appears but occurs without closing of 
the bulk band gap. The difference in the topological 
phase transitions between these two models has been un- 
derstood as a consequence of the symmetry that con- 
serves the z-component of the total-spin quantum num- 
bers of the system, i.e., the spin-quantized conditioni^ 

However, because the spatial electron correlations can- 
not be treated sufficiently in the DMFT framework, the 
study of their effects on the AFTI states in the BHZ 
model still remains to be an important issue. In this 
paper, we will therefore apply the method of VCA^^"— 
to the BHZ model extended with the on-site Hubbard 
interaction, whereby we can treat the short-range spa- 
tial correlations exactly and calculate the single-particle 
Green function directly. We will thus evaluate the spin 
Chern number (SChN)22r— liSii^ and local magnetic mo- 
ment of the AF order and show that the AF state actually 
has the topologically nontrivial structure, i.e., the AFTI 
state exists, under the spin-quantized condition. We will 
furthermore show that the magnetic transition between 
the TBI and AFTI is of the second order and will demon- 
strate that the topological phase transition between the 
AFTI and AFMI occurs with closing of the bulk band 
gap. 

This paper is organized as follows: In Sec. II, we in- 
troduce the extended BHZ model with the on-site Hub- 
bard interaction and briefiy discuss the method of cal- 
culation. In Sec. HI, we first calculate the properties of 
SChN and single-particle spectrum for the bulk and edge 
states in the nonmagnetic case and confirm that the re- 
sult for SChN can determine the topological phase tran- 
sition point in agreement with the single-particle spec- 
tra. We then show the results for the AF ordered case 
and discuss the property of the topologically nontrivial 
AF order, including the influences on the single-particle 



spectra and local magnetic moments. A summary of the 
paper is given in Sec. IV. 



ing the on-site Hubbard interaction U . The Hamiltonian 
reads 



II. MODEL AND METHOD 



We study the BHZ model^" defined on the two- 
dimensional square lattice with an extension of includ- 
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Ti = HbHZ + "Hint = ^ cliii-jCj + [^ ^ nia-\nia\. 
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where Ci = {cii^,Ci2^,Ciii,Ci2i)'^ and rnaa = cj^aCtacr- 
This Hamiltonian contains two orbitals a = 1,2 and spin 
a =t. i- The off-diagonal term in Tiij, which corresponds 
to the spin-orbit coupling, is illustrated in Fig. [TJ In our 
calculation, we consider the particle-hole symmetric case, 
assuming —ti = t2 = t, — ei = £2 = t, and t' = 0.8t. 
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FIG. 1: (Color online) Schematic illustration of the spin-orbit 
coupling term in the BHZ model. The circles labeled "s" and 
"p" correspond to the orbital a = 1 (s) and orbital a = 2 (p), 
respectively, j, j ± x, and j ±y denote the lattice sites. 



The VGA is a quantum cluster method based on the 
self-enegy functional theory (SFT) ^^''^^ and has been ap- 
plied to many interesting problems that include the com- 
petition between the symmetry-broken phases such as the 
superconducting phase, antiferromagnetic phase, etc. In 
the SFT framework, the problem of calculating the exact 
self-energy becomes a variational problem for the grand 
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FIG. 2: (Color online) Sketch of the tiling used for calculating 
the single-particle spectrum for the edge. The dotted line 
represents the inter-cluster hopping term V. We use the 4-site 
cluster for the bulk and edge states in the VGA calculations. 
To see the edge states, we arrange 10 clusters along the y axis. 



potential functional f2[S] defined as 

r2[S] = Tr ln(G(7i - Y.)-^ + F[S] 



(2) 



where F\n\ is the Legendre transfom of the Luttingcr- 
Ward functional $[G] such that the condition 5F[S\ = 
— TG • (5S is satisfied, and therefore, the condition 
(551[S] = corresponds to the Dyson equation G""'^ = 
Gq ^ — S. Tr in Eq. [5] represents both the sum over the 
Matsubara frequency and trace over the single-particle 
bases, such as momentum, spin, orbital, etc. In partic- 
ular, the self-energy S can be obtained in VGA, where 
we use the reference system consisting of the small clus- 
ters with the single-particle parameters t', which are cho- 
sen so as to satisfy the stationary condition of the grand 
potential functional n[S'(t')]. Furthermore, if we intro- 
duce the Weiss field as a variational parameter, we can 
treat the spontaneous symmetry breaking. In the VGA 
framework, the grand potential functional 51[S'(t')] can 



be rewritten as 



n[T.'{t')] = fl'(t') - Tr ln[G'(t')] 

+ Trln[Go-i-S'(t')]-i 



(3) 



where fl'{t') - Tr ln[G'(t')] corresponds to F[S'{t')] in 
the reference system, and Go is the free propagator of 
the original lattice system. Details of VGA can be found 
in Refs. [sl and [3^. 

In the evaluation of the SChN and single-particle spec- 
trum, we use the cluster perturbation theory (CPT)'^ - to 
calculate the lattice Green function G(t'), which is given 
by 



G(t') 



[Go-i-S'(t')]-i 

[Go-i-G'-'(t') + G'o'(t')-S'(t')]-i 

G'(t')[ i-VG'(t')]"^ 



(4) 



where V is the inter-cluster hopping treated perturba- 
tively. In the calculation of the bulk states, we use the 
supercluster where the clusters are arranged along the x 
and y axes and connected periodically along both direc- 
tions. For the edge states, on the other hand, we use the 
supercluster where the clusters are arranged along the x 
and y axes but connected periodically only along the x 
axis (see Fig. [2). 

The SGhN, which characterizes the topological prop- 
erty in the bulk states, may be written in the Green func- 
tion formalism2^i^i2Si^ as 



N ^ 



•^fii/p 



^^^Jd'kTr[aMk)d,G-\k) 
xG{k)d^G-\k)G{k)dpG^\k)] (5) 



partial differentiation with respect to fc'', and e^i^p is the 
antisymmetric symbol. This quantity is in proportion to 
the spin Hall conductivity and has the quantized value as 
long as the total spin S^^^ of the system is conserved42iM 
In general, the Z2 topological insulator is characterized 
by Z2 invariant, which is well-defined as long as the time- 
reversal symmetry remains. The SGhN on the other hand 
is equivalent to the parity of Z2 invariant on the condition 
of the spin conservation and is applicable to the axial AF 
case where the time-reversal symmetry is broken. 

Here, let us consider the SGhN in the noninteracting 
BHZ model for simplicity. In this case, SGhN can be 
divided into the up- and down-spin parts as A^ = N-^—N^, 
and the spin-polarized Hamiltonian H^ can be written in 
k space in the matrix form as 

?i,(k)=^d,(k)f, (6a) 

i 

diCk) = — 2i'sin kj. x sign(cr) (6b) 

d2(k) = -2i'sin ky (6c) 

d3(k) = —t — 2i(cos kx + cos ky) (6d) 



where fi represents the Pauli matrix for the orbital de- 
grees of freedom. Eq. ([5]) can then be rewritten in terms 
of the cr-spin contributions to the spin Ghern number 



N„ 



1 



-— / d2krf(k).afc,d(k)x4.d(k) (7) 





with di ~ di/\d\. Using Eq. ([7]), we obtain SGhN as 
A^ = 1 with N-^ = -N^ = 1/2, where the sign of SGhN 
is determined from the sign of the spin-orbit interaction 
terms of Eqs. (|6b[) and ([Sc|) and reflects the direction of 
the current carried by the helical edge states. 
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where the notation k — («w,k) is used, d^ denotes the -5-0 
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FIG. 3: (Color online) Single-particle spectra for the bulk 
(left panel) and edge state (middle panel) calculated by CPT. 
Right panel shows the density of states (DOS) for the bulk. 
The characteristic values of U used are: (a) [/ = (TBI), (b) 
U = 2.0i (TBI), (c) U = 4.4:1 (topological phase transition 
point), and (d) U = 6.0t (NMI). 



III. RESULTS OF CALCULATION 

First, we discuss the effects of the on-site Hubbard in- 
teraction U on the BHZ model in the nonmagnetic case. 
Figure [3] shows the results for the single-particle spec- 
tra of the bulk and edge states. At U < 4 At, we find 
that the band gap in the bulk state (which we call the 
bulk gap) and the gapless edge state appear, which char- 
acterize the topologically nontrivial state, and thus we 
confirm that this region U < 4.4t belongs to the TBI 
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FIG. 4: (Color online) Calculated SChN for the nonmagnetic 
state. The vertical dotted line at [/ = 4.4t indicates the 
topological phase transition point between the TBI and NMI 
states. 



state. Then, ai U — AAt, the bulk gap closes and simul- 
taneously the gapless edge state vanishes (bulk-edge cor- 
respondence), indicating that the electron correlation U 
derives the topological phase transition. At U > AAt, the 
bulk gap opens again but no gapless edge state appears, 
and thus the system belongs to the trivial nonmagnetic 
insulator (NMI) state. Interestingly, this phase transition 
occurs with closing the bulk gap and the single-particle 
spectrum for the bulk state acquires the linear dispersion. 
This result has also been observed in the VGA study of 
the Kane-Mele model. Therefore, in general, the closing 
of the bulk gap becomes a signal of the appearance of the 
topological phase transition between the TBI and NMI 
states. 

We note here that, although the single-particle spectra 
for the bulk and edge states are useful for detecting the 
topological phase transition as shown above, we cannot 
verify in this calculation whether the edge state actually 
brings the quantized spin current. Also, it is not easy to 
take into account the possible inhomogeneity of the self- 
energies in the edge states in the VGA calculation. The 
SGhN is free from such problems when we characterize 
the topology of the bulk states in correlated systems, 
which we will show in the following. 

The calculated result for SGhN is shown in Fig. 21 
where we find that, with increasing [/, SGhN discontinu- 
ously changes from 1 to with a divergence at U = 4.4t. 
This behavior corresponds to the bulk-gap closing at 
U = AAt shown in Fig. [31 In general, without any bulk- 
gap closing or anomaly of the self energy, the topologi- 
cal properties of the system do not change ] ^^'^^ There- 
fore, this result also indicates that the topological phase 
transition between the TBI and NMI states occurs si- 
multaneously with the bulk-gap closing. In other words, 
this result supports the validity of our calculation of the 
single-particle spectra for the edge state. Note that our 
result for SGhN does not show a step-function-like be- 
havior from 1 to but rather show a diverging behavior 
deviating from 1 or in the vicinity of the transition 
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FIG. 5; (Color online) Schematic representation of the mag- 
netic ordering in the BHZ model. The spheres labeled "s" 
and "p" correspond to the orbital a = 1 (or s) and orbital 
a = 2 (or p), respectively, and the arrows indicate the spin 
directions. 




FIG. 6: (Color online) Calculated grand potential Q.^ — ^^=0 
(per site) as a function of the Weiss field h at several values 
of U . The value of U used is shown on each curve. The dots 
indicate the stationary points. 



point. This is due to a numerical error, which is however 
not serious for determining the topological phase tran- 
sition point because the divergence of SGhN signals the 
transition point accurately. This method of SGhN can 
thus be used to determine the topological phase transi- 
tion point between two topologically distinct states even 
when the system has the AF order, as we will discuss in 
the following. 

Next, we discuss the magnetic instability in the BHZ 
model induced when we introduce the Hubbard interac- 
tion U . Let us first consider the case at large- C/ limit, 
whereby we can search for candidates of the magnetic in- 
stabilities. Our model in this case may be rewritten by 
the following spin Hamiltonian: 
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FIG. 7: (Color online) Calculated spin Chern number (SChN) 
and local magnetic momentum m^. The vertical dotted line 
at f/=3.87t indicates the topological phase transition point 
between the AFTI and AFMI states. The inset enlarges the 
region near the phase transition points. 



with Sia = ^J2t,t' '^aA^V''^' ^iar' and Orbital indices 
a and /3. This Hamiltonian has the in-plane ferromag- 
netic interaction J^Jf^p coming from the spin-dependent 
anisotropic hopping term due to the spin-orbit coupling, 
and therefore, the spin space in this model does not have 
a full SU(2) symmetry but rather has an Ising-like ax- 
ial anisotropy. In Fig. [Sj we show the spin configu- 
ration which is expected to occur in this Hamiltonian. 
This magnetic order satisfies the spin-quantized condi- 
tion, and therefore, SChN is useful for determining the 
topological phase transition in our BHZ model under the 
magnetic order. No other magnetic orders, such as the 
in-plane orders, than that shown in Fig. [31 have been 
detected, which is in contrast to the Kane-Mele model 
where the in-plane magnetic order occurs at the large U 
limit. Note that the Z2 invariant is not meaningful due 
to the time-reversal symmetry breaking and cannot be 
used for determining the topological phase transition. 

Let us then assume the AF order shown in Fig. [5] and 
use the Weiss field of the form 
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FIG. 8: (Color online) Calculated single-particle spectrum 
for the bulk (left panel) and corresponding DOS with the 
orbital decompositions (right panel) at (a) [7=3. 54t (magnetic 
transition point), (b) 17=3. 87t (topological phase transition 
point), and (c) (7=6.00t (AFMI state). 



with Q = (tt, tt) as the variational parameter in our VGA 
calculations. Figure El shows the calculated result for the 
grand potential as the function of the Weiss field h at 
several values of U. We find that there is a stationary 
point at ft. 7^ for U > 3.54t and thus the AF state ap- 
pears for U > 3.54i. We also calculate the local magnetic 
moment defined by 



mz 



1 

2N 



E 



„^Q-r7j 



CLA'^T" C^> 



(10) 



where (• • • ) represents the ground-state expectation 
value, N is the number of the sites in the supercluster, 
and Q = (tt, tt). 

The calculated result for mz is shown in Fig. [71 to- 
gether with the result for SChN. We find that the mag- 
netic transition occurs at [/ = 3.54t, which is of the sec- 
ond order (contrary to the previous DMFT study??), and 
that the SChN has the discontinuous change from 1 to 
with the divergence at U = 3.87t. We thus detect 
the region, i.e., 3.54t < U < 3.87i, where the topologi- 
cally nontrivial state coexists with the AF order, which 
is nothing but the AFTI state. 

Figure [8] shows the results for the single-particle spec- 
tra for the bulk state with the axial AF order at several 
values of U/t (= 3.54,3.87,6.00). We find that the be- 
haviors similar to the nonmagnetic case (see Fig. [3]) are 
observed also in the magnetically ordered case. We in 
particular confirm that the closing of the bulk gap is ob- 
served atU — 3.87t (see Fig.[5Jb)), i.e., at the topological 
phase transition point between the two topologically dis- 
tinct phases, AFTI and AFMI. In the Kane-Mele model, 
it is known that the bulk gap does not close at the tran- 
sition point between TBI and AFMI and that there is 
no coexistence region between the topological order and 



in-plane AF order. Our results therefore indicate the im- 
portance of the the spin quantized condition, in relation 
to the symmetry in spin space, the coexistence region, 
and the bulk band gap. 



IV. SUMMARY 

We have studied the effects of electron correlations 
on the topological phase transitions, focusing in partic- 
ular on the presence of the antiferromagnetic topologi- 
cal insulator state. We have used the Bernevig-Hughes- 
Zhang (BHZ) model with the on-site Hubbard interaction 
and have applied the variational cluster approach (VGA) 
whereby the short-range spatial correlations are taken 
into account exactly. We have calculated the spin Chern 
number (SChN) and local magnetic moment as well as 
the single-particle spectra for the bulk and edge states. 

In the nonmagnetic case, we found that the topological 
phase transition between the topological band insulator 
(TBI) and nonmagnetic insulator (NMI) occurs with the 
simultaneous vanishing of the bulk band gap and gap- 
less edge state. This result may be a general conclusion 
equally valid for other topological phase transitions be- 



tween TBI and NMI. We have also studied the magnetic 
instability of this model and have clarified the relation 
between the topological and antiferromagnetic (AF) or- 
ders. We have calculated the spin Chern number and the 
local magnetic moment of the AF ordered state and con- 
firmed that the magnetic transition, which was found to 
be of the second order, and topological phase transition 
do not occur simultaneously, but rather the topological 
order coexists with the AF order in the spin-quantized 
condition. From the single-particle gap for the bulk state, 
we have demonstrated that the topological phase transi- 
tion between the AF topological insulator state and AF 
Mott insulator state occurs when the bulk gap closes. 
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